Thinking and discussions for designing a testing framework & test runner for Zinnia modules. See https://github.com/filecoin-station/zinnia/issues/44

Sources for inspiration

• https://github.com/jamesshore/ai_chronicles/commit/744e3f39af78ede64181d898f5879d55cf34a094#diff-a8bd36f99429410727335c5d08c079195a0023ba16a8af19a8206c03541b1e5d

• Deno testing: https://deno.com/manual@v1.30.3/basics/testing

• Deno Behaviour-Driven Development: https://deno.com/manual@v1.30.3/basics/testing/behavior_driven_development
  https://github.com/denoland/deno_std/blob/0.177.0/testing/bdd.ts

Why are we building our own testing framework?

When the testing framework is an integral part of the runtime, it can leverage low-level runtime features to provide better developer experience.

For example, deno test can detect when a test case starts an async operation that “leaks”, i.e. does not complete before the test case finish and continues running. If such async operation fails, the error (promise rejected) is detected when a different test case is running. With test runners like Mocha, such errors are reported as if the other test case failed. That’s confusing and it’s difficult to find the test cause that caused the problem.

Another example is code coverage. Historically, tools like “istanbul” used code instrumentation to enable tracing. Setting up the test framework with the code coverage tool is cumbersome. When the test runner is a part of the runtime, it can provide better integration out of the box. Recently, V8 added built-in support for collecting code coverage data. Again, this is easier to use from within the runtime.

⛔️ CONTENT BELOW THIS LINE IS OUTDATED ⛔️

Proposed DX

Running the tests

Running all tests in the current project:

❯ zinnia test

Running all tests in a single file:

❯ zinnia test path/to/file.js

Running all tests matching a given pattern:

❯ zinnia test --filter "a substring or a regexp"

(We can incrementally add more bells and whistles later.)

Writing the tests

I want to support Behaviour-Driven-Development style of testing, as known from Mocha. I also want all testing APIs to be locally scoped, no globals please!

import { describe, it, beforeAll, afterAll, beforeEach, afterEach } from 'zinnia:test';

describe('Some entity', () => {
  // all hooks support both sync and async functions

  // run some code only once per `describe()` block
  beforeAll(() => { /* do some setup - only once */ });
  afterAll(() => { /* teardown after the test suite finished */ });

  // run some code for each test
  beforeEach(() => { /* run some setup before each test */ });
  afterEach(() => { /* cleanup after each test */ });

  it('does something under some conditions', () => {
    // a test can be sync
  });

  it('does something else under different conditions', async () => {
   // a test can be also async
  });

  describe('constructor', () => {
    // describe blocks can be nested
  })
});

describe('Another entity', () => {
  // there can be more than one describe block per file
});

// async describe blocks are not permitted,
// all test suites must be sync.
// We can add support for async test suites later if needed.
describe('this will throw', async () => {
  // oops
});

// The top-level `describe()` block is optional,
// each file behaves as a virtual `describe()` block
// Top-level hooks are executed only when we execute
// at least one test from the file.
beforeAll(() => { /* per-file setup */ })
it('does something', () => { /*...*/ })

We should add .only and .skip modifiers soon. It’s important to implement .only ergonomically, so that adding .only to a nested it will run only this particular test case, even if we don’t add .only to the parent describe blocks. This may seem obvious, but it’s not supported by popular test frameworks like Deno BDD and node:test.

Walking skeleton

1. Running tests:
   1. Run tests in a single file: zinnia test path/to/file.js
   2. Exit with a non-zero code when some tests failed.

2. Writing tests:
   1. Only two supported APIs: describe and it
   2. After a test fails, we still execute other tests in the suite.
   3. Test results are reported in a user-friendly way.

3. Initial documentation for module builders

I don’t think we can avoid implementing the item (1) ourselves.

There may be ways how to leverage existing JS frameworks as a shortcut to get us to (2), although this would make it more difficult to add advanced features requiring deep integration with runtime. However:

• I prefer to NOT become too tied to details of any particular framework, e.g. what is returned by it call, in which order are suites, test cases and hooks invoked, etc.

• I want to avoid globals. They are adding too much complexity to linting and type checking, plus make it more difficult to get a nice IDE integration (auto-completion support, etc.)

Run all tests

Run all tests in the project: zinnia test

Further improvents

Incremental improvements, can be implemented in any order:

More incremental improvements that go beyond the scope of the initial release: